Thank you @TingLei-NOAA and @emilyhcliu for your quick reviews.

@xyzemc , now that we have two peer review approvals, we need to run regression (ctests) on WCOSS2, Hera, and Orion. Would you please do so?

Thank you @TingLei-NOAA and @emilyhcliu for your quick reviews.

@xyzemc , now that we have two peer review approvals, we need to run regression (ctests) on WCOSS2, Hera, and Orion. Would you please do so?

Yes, I will run the regression test.

@xyzemc , we can merge this PR into develop once regression test results are recorded in this PR or the originating issue (#648)

@xyzemc , we can merge this PR into develop once regression test results are recorded in this PR or the originating issue (#648)

I have completed the regression test on Hera. Should I post the test result here? It is located at /scratch1/NCEPDEV/da/Xiaoyan.Zhang/noscrub/regression

Thank you @xyzemc for running the Hera regression test. Yes, please post the results in this PR or in the originating issue, #648.

Oops, @xyzemc , when I said post the ctest results I was only referring to the ctest output. For example, look in PR #669. Gang cut-n-pasted the ctest output under the How Has This Been Tested? section of the PR. Click here to see a different example from PR #654.

The regression tests on Hera passed:
[Xiaoyan.Zhang@hfe05 build]$ ctest -R global

Test project /scratch1/NCEPDEV/stmp2/Xiaoyan.Zhang/regrssion/develop/build

1/2 Test #1: [=[global_4denvar]=] ............. Passed 4016.24 sec
Start 7: [=[global_enkf]=]
2/2 Test #7: [=[global_enkf]=] ................ Passed 2338.15 sec

100% tests passed, 0 tests failed out of 2

Total Test time (real) = 6354.41 sec

@xyzemc , why are only the global ctests being run (ctest -R global)? Some of regional tests assimilate radiances. The easiest thing to do is to run all tests via ctest -j 7

@xyzemc , why are only the global ctests being run (ctest -R global)? Some of regional tests assimilate radiances. The easiest thing to do is to run all tests via ctest -j 7

Thanks! I will do all the ctest on Hera, wcoss, and orion today.

The regression tests on wcoss2 passed:
build>ctest -j7
Test project /lfs/h2/emc/ptmp/Xiaoyan.Zhang/develop/build
Start 4: netcdf_fv3_regional
Start 1: global_4denvar
Start 5: hafs_4denvar_glbens
Start 6: hafs_3denvar_hybens
Start 7: global_enkf
Start 2: rtma
Start 3: rrfs_3denvar_glbens
1/7 Test #4: netcdf_fv3_regional .............. Passed 483.49 sec
2/7 Test #3: rrfs_3denvar_glbens .............. Passed 546.02 sec
3/7 Test #2: rtma ............................. Passed 970.33 sec
4/7 Test #7: global_enkf ...................... Passed 1119.09 sec
5/7 Test #6: hafs_3denvar_hybens .............. Passed 1216.48 sec
6/7 Test #5: hafs_4denvar_glbens .............. Passed 1333.01 sec
7/7 Test #1: global_4denvar ................... Passed 1493.54 sec

100% tests passed, 0 tests failed out of 7

Total Test time (real) = 1493.65 sec

The regression tests on Hera passed:
[Xiaoyan.Zhang@hfe04 build]$ ctest -j7
Test project /scratch1/NCEPDEV/stmp2/Xiaoyan.Zhang/regrssion/develop/build
Start 5: [=[hafs_4denvar_glbens]=]
Start 7: [=[global_enkf]=]
Start 1: [=[global_4denvar]=]
Start 6: [=[hafs_3denvar_hybens]=]
Start 2: [=[rtma]=]
Start 3: [=[rrfs_3denvar_glbens]=]
Start 4: [=[netcdf_fv3_regional]=]
1/7 Test #3: [=[rrfs_3denvar_glbens]=] ........ Passed 2067.16 sec
2/7 Test #4: [=[netcdf_fv3_regional]=] ........ Passed 2180.45 sec
3/7 Test #7: [=[global_enkf]=] ................ Passed 2506.86 sec
4/7 Test #2: [=[rtma]=] ....................... Passed 2600.76 sec
5/7 Test #6: [=[hafs_3denvar_hybens]=] ........ Passed 2674.82 sec
6/7 Test #5: [=[hafs_4denvar_glbens]=] ........ Passed 2914.65 sec
7/7 Test #1: [=[global_4denvar]=] ............. Passed 3000.58 sec

100% tests passed, 0 tests failed out of 7

Total Test time (real) = 3000.60 sec

The regression tests on Orion:
Test project /work/noaa/fv3-cam/xyzhang/develop/build
Start 1: [=[global_4denvar]=]
Start 5: [=[hafs_4denvar_glbens]=]
Start 2: [=[rtma]=]
Start 6: [=[hafs_3denvar_hybens]=]
Start 3: [=[rrfs_3denvar_glbens]=]
Start 7: [=[global_enkf]=]
Start 4: [=[netcdf_fv3_regional]=]
1/7 Test #7: [=[global_enkf]=] ................***Failed 488.40 sec
2/7 Test #4: [=[netcdf_fv3_regional]=] ........***Failed 605.43 sec
3/7 Test #3: [=[rrfs_3denvar_glbens]=] ........***Failed 728.33 sec
4/7 Test #2: [=[rtma]=] ....................... Passed 1036.78 sec
5/7 Test #6: [=[hafs_3denvar_hybens]=] ........ Passed 1457.59 sec
6/7 Test #1: [=[global_4denvar]=] ............. Passed 1682.34 sec
7/7 Test #5: [=[hafs_4denvar_glbens]=] ........***Failed 1817.42 sec

43% tests passed, 4 tests failed out of 7

Total Test time (real) = 1817.56 sec

The following tests FAILED:
3 - [=[rrfs_3denvar_glbens]=] (Failed)
4 - [=[netcdf_fv3_regional]=] (Failed)
5 - [=[hafs_4denvar_glbens]=] (Failed)
7 - [=[global_enkf]=] (Failed)
Errors while running CTest
The Test 7 global_enkf failed due to a runtime check

The runtime for global_enkf_loproc_updat is 86.625029 seconds and is within the maximum allowable operational time of 1200 seconds,
continuing with regression test.
The runtime for global_enkf_loproc_updat is 86.625029 seconds and is within the allowable threshold time of 94.983513 seconds,
continuing with regression test.
The runtime for global_enkf_hiproc_updat is 65.267664 seconds. This has exceeded maximum allowable threshold time of 64.598262 seconds, resulting in Failure timethresh2 of the regression test.
The memory for global_enkf_loproc_updat is 3652352 KBs and is within the maximum allowable memory of 4087291 KBs,
continuing with regression test.
The results between the two runs (global_enkf_loproc_updat and global_enkf_loproc_contrl) are reproducible.
The results between the two runs (global_enkf_loproc_updat and global_enkf_hiproc_updat) are reproducible

Output from these tests are in: /work/noaa/fv3-cam/xyzhang/tmpreg_global_enkf/tmpreg_global_enkf/global_enkf_loproc_updat_vs_global_enkf_loproc_contrl/global_enkf_regression_results.txt

The Test 4 netcdf_fv3_regional failed due to a runtime check

The runtime for netcdf_fv3_regional_loproc_updat is 142.932236 seconds and is within the maximum allowable operational time of 1200 seconds,continuing with regression test.

The runtime for netcdf_fv3_regional_loproc_updat is 142.932236 seconds. This has exceeded maximum allowable threshold time of 100.412602 seconds, resulting in Failure time-thresh of the regression test.

The runtime for netcdf_fv3_regional_hiproc_updat is 64.615032 seconds and is within the allowable threshold time of 80.413887 seconds, continuing with regression test.

The memory for netcdf_fv3_regional_loproc_updat is 308000 KBs and is within the maximum allowable memory of 341000 KBs, continuing with regression test.

The results (penalty) between the two runs (netcdf_fv3_regional_loproc_updat and netcdf_fv3_regional_loproc_contrl) are reproducible.

The results (penalty) between the two runs (netcdf_fv3_regional_loproc_updat and netcdf_fv3_regional_hiproc_updat) are reproducible

The test results is in: /work/noaa/fv3-cam/xyzhang/tmpreg_netcdf_fv3_regional/tmpreg_netcdf_fv3_regional/netcdf_fv3_regional_loproc_updat_vs_netcdf_fv3_regional_loproc_contrl/netcdf_fv3_regional_regression_results.txt

The Test 3 rrfs_3denvar_glbens failed due to a runtime check

The runtime for rrfs_3denvar_glbens_loproc_updat is 220.475881 seconds and is within the maximum allowable operational time of 1200 seconds,
continuing with regression test.

The runtime for rrfs_3denvar_glbens_loproc_updat is 220.475881 seconds. This has exceeded maximum allowable threshold time of 184.339221 seconds,
resulting in Failure time-thresh of the regression test.

The runtime for rrfs_3denvar_glbens_hiproc_updat is 73.899902 seconds and is within the allowable threshold time of 117.254622 seconds,
continuing with regression test.

The memory for rrfs_3denvar_glbens_loproc_updat is 1995820 KBs and is within the maximum allowable memory of 2231451 KBs,
continuing with regression test.

The results (penalty) between the two runs (rrfs_3denvar_glbens_loproc_updat and rrfs_3denvar_glbens_loproc_contrl) are reproducible.

The fv3_dynvars are reproducible
The fv3_sfcdata are reproducible
The fv3_tracer are reproducible
The results between the two runs (rrfs_3denvar_glbens_loproc_updat and rrfs_3denvar_glbens_loproc_contrl) are reproducible
since the corresponding results are identical.

The results (penalty) between the two runs (rrfs_3denvar_glbens_loproc_updat and rrfs_3denvar_glbens_hiproc_updat) are reproducible

The fv3_dynvars are reproducible
The fv3_sfcdata are reproducible
The fv3_tracer are reproducible
The results between the two runs (rrfs_3denvar_glbens_loproc_updat and rrfs_3denvar_glbens_hiproc_updat) are reproducible
since the corresponding results are identical.

The test results is in: /tmpreg_rrfs_3denvar_glbens/rrfs_3denvar_glbens_loproc_updat_vs_rrfs_3denvar_glbens_loproc_contrl/rrfs_3denvar_glbens_regression_results.txt

The Test 5 rrfs_3denvar_glbens failed due to a runtime check

The runtime for rrfs_3denvar_glbens_loproc_updat is 220.475881 seconds. This has exceeded maximum allowable threshold time of 184.339221 seconds,
resulting in Failure time-thresh of the regression test.

The runtime for rrfs_3denvar_glbens_hiproc_updat is 73.899902 seconds and is within the allowable threshold time of 117.254622 seconds,
continuing with regression test.

The memory for rrfs_3denvar_glbens_loproc_updat is 1995820 KBs and is within the maximum allowable memory of 2231451 KBs,
continuing with regression test.

The results (penalty) between the two runs (rrfs_3denvar_glbens_loproc_updat and rrfs_3denvar_glbens_loproc_contrl) are reproducible.

The fv3_dynvars are reproducible
The fv3_sfcdata are reproducible
The fv3_tracer are reproducible
The results between the two runs (rrfs_3denvar_glbens_loproc_updat and rrfs_3denvar_glbens_loproc_contrl) are reproducible
since the corresponding results are identical.

The results (penalty) between the two runs (rrfs_3denvar_glbens_loproc_updat and rrfs_3denvar_glbens_hiproc_updat) are reproducible

The fv3_dynvars are reproducible
The fv3_sfcdata are reproducible
The fv3_tracer are reproducible
The results between the two runs (rrfs_3denvar_glbens_loproc_updat and rrfs_3denvar_glbens_hiproc_updat) are reproducible
since the corresponding results are identical.

The test results is in: /work/noaa/fv3-cam/xyzhang/tmpreg_rrfs_3denvar_glbens/tmpreg_rrfs_3denvar_glbens/rrfs_3denvar_glbens_loproc_updat_vs_rrfs_3denvar_glbens_loproc_contrl/rrfs_3denvar_glbens_regression_results.txt

Thank you @xyzemc for running regression tests on WCOSS2, Hera, and Orion. The Orion failures are not fatal. Given the two peer approvals and documented ctests, I will forward this PR to the Handling Review Team for action.